We present SODA: the first publicly available, million-scale high-quality social dialogue dataset. Using SODA, we train COSMO: a generalizable conversation agent outperforming previous best-performing agents on both in- and out-of-domain datasets. In contrast to most existing crowdsourced, small-scale dialogue corpora, we distill 1.5M socially-grounded dialogues from a pre-trained language model (InstructGPT; Ouyang et al., 2022). Dialogues are distilled by contextualizing social commonsense knowledge from a knowledge graph (Atomic10x; West et al., 2022). Human evaluation shows that dialogues in SODA are more consistent, specific, and (surprisingly) natural than prior human-authored datasets - e.g., DailyDialog (Li et al., 2017), BlendedSkillTalk (Smith et al., 2020). In addition, extensive evaluations show that COSMO is significantly more natural and consistent on unseen datasets than best-performing dialogue models - e.g., GODEL (Peng et al., 2022), BlenderBot (Roller et al., 2021), DialoGPT (Zhang et al., 2020). Furthermore, it is sometimes even preferred to the original human-written gold responses. We make our data, models, and code public.

我们挑战AI模型，以“展示”对《纽约客》标题比赛的复杂多模式幽默的理解。具体而言，我们开发了三个精心限制的任务，以掌握图像和标题之间的潜在复杂和意外的关系，并且对人类经验的广泛品种产生了复杂和意外的寓意；这些是纽约口径卡通的标志。我们调查了直接将卡通像素和字幕输入的视觉和语言模型，以及仅通过提供图像的文本描述来规避图像处理的仅限语言模型。即使我们为卡通图像提供了丰富的多方面注释，我们也可以确定高质量的机器学习模型（例如，微调，175b参数语言模型）和人类之间的性能差距。我们公开发布我们的语料库，包括描述图像的位置/实体的注释，场景的不寻常以及对笑话的解释。

人类具有出色的能力来推理绑架并假设超出图像的字面内容的内容。通过识别散布在整个场景中的具体视觉线索，我们几乎不禁根据我们的日常经验和对世界的知识来提出可能的推论。例如，如果我们在道路旁边看到一个“ 20英里 /小时”的标志，我们可能会假设街道位于居民区（而不是在高速公路上），即使没有房屋。机器可以执行类似的视觉推理吗？我们提出了Sherlock，这是一个带注释的103K图像的语料库，用于测试机器能力，以超出字面图像内容的绑架推理。我们采用免费观看范式：参与者首先观察并识别图像中的显着线索（例如，对象，动作），然后给定线索，然后提供有关场景的合理推论。我们总共收集了363K（线索，推理）对，该对形成了首个绑架的视觉推理数据集。使用我们的语料库，我们测试了三个互补的绑架推理轴。我们评估模型的能力：i）从大型候选人语料库中检索相关推论； ii）通过边界框来定位推论的证据，iii）比较合理的推论，以匹配人类在新收集的19k李克特级判断的诊断语料库上的判断。尽管我们发现具有多任务目标的微调夹RN50x64优于强大的基准，但模型性能与人类一致之间存在着重要的净空。可在http://visualabduction.com/上获得数据，模型和排行榜

作为人类，我们通过我们所有的感官来驾驭世界，使用每个人从每个人纠正其他人。我们介绍了Merlot Reserve，一个模型，该模型是联合随着时间的推移而表示视频的模型 - 通过从音频，字幕和视频帧学习的新培训目标。给出了一个视频，我们用掩模令牌替换文本和音频的片段;该模型通过选择正确的蒙版片段来学习。我们的目标比替代方面更快地学习，并在规模上表现良好：我们预先逼近2000万YouTube视频。经验结果表明，Merlot Reserve学会通过所有组成模式的视频的强烈陈述。在FineTuned时，它在VCR和TVQA上为VCR和TVQA进行了新的最先进，优先于前勤工作分别为5％和7％。消融表明，两个任务都受益于音频预制 - 甚至录像机，围绕图像中心的QA任务（没有声音）。此外，我们的客观使开箱即用的预测，揭示了强大的多式联合致辞理解。在一个完全零拍摄的环境中，我们的模型在四个视频理解任务中获得竞争结果，甚至优于最近提出的定位推理（星）基准的监督方法。我们分析为什么包含音频导致更好的视觉语言表示，这表明未来研究的重要机会。我们通过讨论多式联运预测的道德和社会影响来得出结论。

可以代表和描述环境声音的机器具有实际潜力，例如，用于音频标记和标题系统。普遍的学习范式已经依赖于并行音频文本数据，但是，Web上几乎没有可用。我们提出了vip-ant，它在不使用任何并行音频文本数据的情况下诱导\ textbf {a} udio- \ textBF {t} EXT对齐。我们的主要思想是在双模形图像文本表示和双模态图像 - 音频表示之间共享图像模型;图像模态用作枢轴，并将音频和文本连接在三模态嵌入空间中。在没有配对的音频文本数据的困难零拍设置中，我们的模型在ESC50和US8K音频分类任务上展示了最先进的零点性能，甚至超过了披肩标题的领域的监督状态检索（带音频查询）2.2 \％R @ 1。我们进一步调查了最小音频监控的情况，发现，例如，只有几百个监督的音频文本对将零拍音频分类精度提高8 \％US8K。然而，为了匹配人类奇偶校验，我们的经验缩放实验表明我们需要大约2米$ 2 ^ {21} \约2M $监督的音频标题对。我们的工作开辟了新的途径，用于学习音频文本连接，几乎没有并行音频文本数据。

大型语言模型越来越能够通过相对较少的特定任务的监督产生流畅的出现文本。但这些模型可以准确解释分类决策吗？我们考虑使用少量人写的例子（即，以几滴方式）生成自由文本解释的任务。我们发现（1）创作更高质量的例子，以提示导致更高质量的世代; （2）令人惊讶的是，在头到头比较中，人群公司通常更喜欢GPT-3生成的解释，以众包中包含的人性写入的解释。然而，Crowdworker评级也表明，虽然模型产生了事实，语法和充分的解释，但它们具有改进的空间，例如沿着提供新颖信息和支持标签的轴。我们创建了一种管道，该管道将GPT-3与监督过滤器结合起来，该过滤器通过二进制可接受性判断来包含人类循环。尽管具有重要的主观性内在的判断可接受性，但我们的方法能够始终如一地过滤人类可接受的GPT-3生成的解释。

The common practice for training commonsense models has gone from-human-to-corpus-to-machine: humans author commonsense knowledge graphs in order to train commonsense models. In this work, we investigate an alternative, from-machine-to-corpus-to-machine: general language models author these commonsense knowledge graphs to train commonsense models. Our study leads to a new framework, Symbolic Knowledge Distillation. As with prior art in Knowledge Distillation (Hinton et al., 2015), our approach uses larger models to teach smaller models. A key difference is that we distill knowledge symbolically-as text-in addition to the neural model. We also distill only one aspect-the commonsense of a general language model teacher, allowing the student to be a different type, a commonsense model. Altogether, we show that careful prompt engineering and a separately trained critic model allow us to selectively distill high-quality causal commonsense from GPT-3, a general language model. Empirical results demonstrate that, for the first time, a human-authored commonsense knowledge graph is surpassed by our automatically distilled variant in all three criteria: quantity, quality, and diversity. In addition, it results in a neural commonsense model that surpasses the teacher model's commonsense capabilities despite its 100x smaller size. We apply this to the ATOMIC resource, and share our new symbolic knowledge graph and commonsense models.

Traditionally, data analysis and theory have been viewed as separate disciplines, each feeding into fundamentally different types of models. Modern deep learning technology is beginning to unify these two disciplines and will produce a new class of predictively powerful space weather models that combine the physical insights gained by data and theory. We call on NASA to invest in the research and infrastructure necessary for the heliophysics' community to take advantage of these advances.

Regularising the parameter matrices of neural networks is ubiquitous in training deep models. Typical regularisation approaches suggest initialising weights using small random values, and to penalise weights to promote sparsity. However, these widely used techniques may be less effective in certain scenarios. Here, we study the Koopman autoencoder model which includes an encoder, a Koopman operator layer, and a decoder. These models have been designed and dedicated to tackle physics-related problems with interpretable dynamics and an ability to incorporate physics-related constraints. However, the majority of existing work employs standard regularisation practices. In our work, we take a step toward augmenting Koopman autoencoders with initialisation and penalty schemes tailored for physics-related settings. Specifically, we propose the "eigeninit" initialisation scheme that samples initial Koopman operators from specific eigenvalue distributions. In addition, we suggest the "eigenloss" penalty scheme that penalises the eigenvalues of the Koopman operator during training. We demonstrate the utility of these schemes on two synthetic data sets: a driven pendulum and flow past a cylinder; and two real-world problems: ocean surface temperatures and cyclone wind fields. We find on these datasets that eigenloss and eigeninit improves the convergence rate by up to a factor of 5, and that they reduce the cumulative long-term prediction error by up to a factor of 3. Such a finding points to the utility of incorporating similar schemes as an inductive bias in other physics-related deep learning approaches.

We introduce camouflaged data poisoning attacks, a new attack vector that arises in the context of machine unlearning and other settings when model retraining may be induced. An adversary first adds a few carefully crafted points to the training dataset such that the impact on the model's predictions is minimal. The adversary subsequently triggers a request to remove a subset of the introduced points at which point the attack is unleashed and the model's predictions are negatively affected. In particular, we consider clean-label targeted attacks (in which the goal is to cause the model to misclassify a specific test point) on datasets including CIFAR-10, Imagenette, and Imagewoof. This attack is realized by constructing camouflage datapoints that mask the effect of a poisoned dataset.